{
  "cells": [
    {
      "cell_type": "markdown",
      "id": "c919f307-07b1-41bd-bc5d-51edd8677983",
      "metadata": {
        "id": "c919f307-07b1-41bd-bc5d-51edd8677983"
      },
      "source": [
        "<div align=\"center\">\n",
        "<h2>Building Data Ingestion from Scratch</h2>\n",
        "</div>\n",
        "\n",
        "<div align=\"center\">\n",
        "    <h3><a href=\"https://aiengineering.academy/\" target=\"_blank\">AI Engineering.academy</a></h3>\n",
        "</div>\n",
        "\n",
        "<div align=\"center\">\n",
        "<a href=\"https://aiengineering.academy/\" target=\"_blank\">\n",
        "<img src=\"https://raw.githubusercontent.com/adithya-s-k/AI-Engineering.academy/main/assets/banner.png\" alt=\"AI Engineering Academy\" width=\"50%\">\n",
        "</a>\n",
        "</div>\n",
        "\n",
        "<div align=\"center\">\n",
        "\n",
        "[![GitHub Stars](https://img.shields.io/github/stars/adithya-s-k/AI-Engineering.academy?style=social)](https://github.com/adithya-s-k/AI-Engineering.academy/stargazers)\n",
        "[![GitHub Forks](https://img.shields.io/github/forks/adithya-s-k/AI-Engineering.academy?style=social)](https://github.com/adithya-s-k/AI-Engineering.academy/network/members)\n",
        "[![GitHub Issues](https://img.shields.io/github/issues/adithya-s-k/AI-Engineering.academy)](https://github.com/adithya-s-k/AI-Engineering.academy/issues)\n",
        "[![GitHub Pull Requests](https://img.shields.io/github/issues-pr/adithya-s-k/AI-Engineering.academy)](https://github.com/adithya-s-k/AI-Engineering.academy/pulls)\n",
        "[![License](https://img.shields.io/github/license/adithya-s-k/AI-Engineering.academy)](https://github.com/adithya-s-k/AI-Engineering.academy/blob/main/LICENSE)\n",
        "\n",
        "</div>\n",
        "\n",
        "\n",
        "## Introduction\n",
        "\n",
        "Data ingestion is a crucial first step in building effective Retrieval-Augmented Generation (RAG) systems. It involves the process of collecting, processing, and storing data in a format that can be efficiently retrieved and used by the RAG model. This README provides an overview of the data ingestion process for RAG systems.\n",
        "\n",
        "## Importance of Data Ingestion in RAG\n",
        "\n",
        "Effective data ingestion is essential for RAG systems because it:\n",
        "1. Determines the quality and relevance of information available for retrieval.\n",
        "2. Affects the system's ability to understand and process queries accurately.\n",
        "3. Impacts the overall performance and efficiency of the RAG pipeline.\n",
        "4. Enables the system to handle diverse data sources and formats.\n",
        "\n",
        "## Key Steps in Data Ingestion\n",
        "\n",
        "The data ingestion process typically involves the following steps:\n",
        "\n",
        "```mermaid\n",
        "flowchart TB\n",
        "    A[Data Collection] --> B[Data Cleaning]\n",
        "    B --> C[Document Splitting]\n",
        "    C --> D[Metadata Extraction]\n",
        "    D --> E[Embedding Generation]\n",
        "    E --> F[Indexing and Storage]\n",
        "```\n",
        "\n",
        "1. **Data Collection**: Gathering information from various sources such as databases, APIs, web scraping, or file systems.\n",
        "\n",
        "2. **Data Cleaning**: Preprocessing the collected data to remove noise, handle missing values, and standardize formats.\n",
        "\n",
        "3. **Document Splitting**: Breaking down large documents into smaller, manageable chunks for more effective retrieval.\n",
        "\n",
        "4. **Metadata Extraction**: Identifying and extracting relevant metadata from the documents to enhance retrieval capabilities.\n",
        "\n",
        "5. **Embedding Generation**: Creating vector representations of the text chunks to enable semantic search.\n",
        "\n",
        "6. **Indexing and Storage**: Organizing and storing the processed data in a format optimized for quick retrieval, often using vector databases or search engines.\n",
        "\n",
        "## Challenges in Data Ingestion\n",
        "\n",
        "- Handling diverse data formats and sources\n",
        "- Ensuring data quality and consistency\n",
        "- Managing large volumes of data efficiently\n",
        "- Updating and maintaining the knowledge base\n",
        "- Balancing between chunk size and semantic coherence\n",
        "\n",
        "## Best Practices\n",
        "\n",
        "1. **Data Quality**: Implement robust data cleaning and validation processes.\n",
        "2. **Scalability**: Design the ingestion pipeline to handle growing data volumes.\n",
        "3. **Metadata Enrichment**: Extract and store relevant metadata to improve retrieval accuracy.\n",
        "4. **Incremental Updates**: Develop mechanisms for efficiently updating the knowledge base.\n",
        "5. **Monitoring**: Implement logging and monitoring to track ingestion performance and data quality.\n",
        "\n",
        "## Conclusion\n",
        "\n",
        "A well-designed data ingestion process is fundamental to the success of a RAG system. It ensures that the information retrieved is accurate, relevant, and up-to-date, ultimately leading to better-quality responses from the language model.\n",
        "\n",
        "In the following sections, we'll explore other crucial components of RAG systems, including data chunking, embedding generation, and retrieval mechanisms."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "92b20306",
      "metadata": {
        "id": "92b20306"
      },
      "source": [
        "If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7ae74e61",
      "metadata": {
        "id": "7ae74e61"
      },
      "outputs": [],
      "source": [
        "!!pip install llama-index\n",
        "!pip install llama-index-llms-openai\n",
        "!pip install llama-index-embeddings-openai\n",
        "!pip install llama-index-vector-stores-qdrant\n",
        "!pip -q install python-dotenv \n",
        "!pip install -U qdrant_client fastembed\n",
        "!pip install pymupdf"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "22fb9e0a-566b-4f34-b9cf-72193cb51adb",
      "metadata": {
        "id": "22fb9e0a-566b-4f34-b9cf-72193cb51adb"
      },
      "source": [
        "## OpenAI\n",
        "\n",
        "You will need an [OpenAI](https://openai.com/) api key for this tutorial. Login to your [platform.openai.com](https://platform.openai.com/) account, click on your profile picture in the upper right corner, and choose 'API Keys' from the menu. Create an API key for this tutorial and save it. You will need it below."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "PWMbn7GooMm5",
      "metadata": {
        "id": "PWMbn7GooMm5"
      },
      "source": [
        "Set your OpenAI api key, and Pinecone api key and environment in the file we created."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "QOyfIoXAoVGX",
      "metadata": {
        "id": "QOyfIoXAoVGX"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from dotenv import load_dotenv\n",
        "\n",
        "load_dotenv()"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "493cfa6d",
      "metadata": {},
      "source": [
        "## Setting up Vector Database\n",
        "\n",
        "We will be using qDrant as the Vector database\n",
        "There are 4 ways to initialize qdrant \n",
        "\n",
        "1. Inmemory\n",
        "```python\n",
        "client = qdrant_client.QdrantClient(location=\":memory:\")\n",
        "```\n",
        "2. Disk\n",
        "```python\n",
        "client = qdrant_client.QdrantClient(path=\"./data\")\n",
        "```\n",
        "3. Self hosted or Docker\n",
        "```python\n",
        "\n",
        "client = qdrant_client.QdrantClient(\n",
        "    # url=\"http://<host>:<port>\"\n",
        "    host=\"localhost\",port=6333\n",
        ")\n",
        "```\n",
        "\n",
        "4. Qdrant cloud\n",
        "```python\n",
        "client = qdrant_client.QdrantClient(\n",
        "    url=QDRANT_CLOUD_ENDPOINT,\n",
        "    api_key=QDRANT_API_KEY,\n",
        ")\n",
        "```\n",
        "\n",
        "for this notebook we will be using qdrant cloud"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0f2746a9",
      "metadata": {},
      "outputs": [],
      "source": [
        "import qdrant_client\n",
        "\n",
        "# LlamaIndex core imports\n",
        "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader\n",
        "from llama_index.core import Settings\n",
        "\n",
        "# LlamaIndex vector store import\n",
        "from llama_index.vector_stores.qdrant import QdrantVectorStore\n",
        "\n",
        "# creating a qdrant client instance\n",
        "\n",
        "client = qdrant_client.QdrantClient(\n",
        "    # you can use :memory: mode for fast and light-weight experiments,\n",
        "    # it does not require to have Qdrant deployed anywhere\n",
        "    # but requires qdrant-client >= 1.1.1\n",
        "    # location=\":memory:\"\n",
        "    # otherwise set Qdrant instance address with:\n",
        "    # url=QDRANT_CLOUD_ENDPOINT,\n",
        "    # otherwise set Qdrant instance with host and port:\n",
        "    host=\"localhost\",\n",
        "    port=6333\n",
        "    # set API KEY for Qdrant Cloud\n",
        "    # api_key=QDRANT_API_KEY,\n",
        "    # path=\"./db/\"\n",
        ")\n",
        "\n",
        "vector_store = QdrantVectorStore(client=client, collection_name=\"01_Data_Ingestion\")"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "be9437a0-3d52-4586-8217-43944971a2cc",
      "metadata": {
        "id": "be9437a0-3d52-4586-8217-43944971a2cc"
      },
      "source": [
        "## Build an Ingestion Pipeline from Scratch\n",
        "\n",
        "We show how to build an ingestion pipeline as mentioned in the introduction.\n",
        "\n",
        "Note that steps (2) and (3) can be handled via our `NodeParser` abstractions, which handle splitting and node creation.\n",
        "\n",
        "For the purposes of this tutorial, we show you how to create these objects manually."
      ]
    },
    {
      "cell_type": "markdown",
      "id": "6d1c9630-2a6b-4656-b272-de1b869c8977",
      "metadata": {
        "id": "6d1c9630-2a6b-4656-b272-de1b869c8977"
      },
      "source": [
        "### 1. Load Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "48739cfc-c05a-420a-8c78-280892f8d7a0",
      "metadata": {
        "id": "48739cfc-c05a-420a-8c78-280892f8d7a0",
        "outputId": "cf6c21a1-d0a7-4b4b-dc40-93ce99824100"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2023-10-13 01:45:14--  https://arxiv.org/pdf/2307.09288.pdf\n",
            "Resolving arxiv.org (arxiv.org)... 128.84.21.199\n",
            "Connecting to arxiv.org (arxiv.org)|128.84.21.199|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 13661300 (13M) [application/pdf]\n",
            "Saving to: ‘data/llama2.pdf’\n",
            "\n",
            "data/llama2.pdf     100%[===================>]  13.03M  7.59MB/s    in 1.7s    \n",
            "\n",
            "2023-10-13 01:45:16 (7.59 MB/s) - ‘data/llama2.pdf’ saved [13661300/13661300]\n"
          ]
        }
      ],
      "source": [
        "!mkdir data\n",
        "!wget --user-agent \"Mozilla\" \"https://arxiv.org/pdf/2307.09288.pdf\" -O \"../data/llama2.pdf\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "079666c5-0685-413d-a765-17f71ae89506",
      "metadata": {
        "id": "079666c5-0685-413d-a765-17f71ae89506"
      },
      "outputs": [],
      "source": [
        "import fitz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "4eee7692-2188-4552-9f2e-cb90ac6b7678",
      "metadata": {
        "id": "4eee7692-2188-4552-9f2e-cb90ac6b7678"
      },
      "outputs": [],
      "source": [
        "file_path = \"../data/llama2.pdf\"\n",
        "doc = fitz.open(file_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "74c573db-1863-45c3-9049-8bad535e6e35",
      "metadata": {
        "id": "74c573db-1863-45c3-9049-8bad535e6e35"
      },
      "source": [
        "### 2. Use a Text Splitter to Split Documents\n",
        "\n",
        "Here we import our `SentenceSplitter` to split document texts into smaller chunks, while preserving paragraphs/sentences as much as possible."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9e175007-84d5-406e-bf5f-6ecacfbfd152",
      "metadata": {
        "id": "9e175007-84d5-406e-bf5f-6ecacfbfd152"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.node_parser import SentenceSplitter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0dbccb26-ea2a-48c9-adb4-1ebe88adaa1c",
      "metadata": {
        "id": "0dbccb26-ea2a-48c9-adb4-1ebe88adaa1c"
      },
      "outputs": [],
      "source": [
        "text_parser = SentenceSplitter(\n",
        "    chunk_size=1024,\n",
        "    # separator=\" \",\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "7a9bed96-adfa-40c9-92bd-9dba68d58730",
      "metadata": {
        "id": "7a9bed96-adfa-40c9-92bd-9dba68d58730"
      },
      "outputs": [],
      "source": [
        "text_chunks = []\n",
        "# maintain relationship with source doc index, to help inject doc metadata in (3)\n",
        "doc_idxs = []\n",
        "for doc_idx, page in enumerate(doc):\n",
        "    page_text = page.get_text(\"text\")\n",
        "    cur_text_chunks = text_parser.split_text(page_text)\n",
        "    text_chunks.extend(cur_text_chunks)\n",
        "    doc_idxs.extend([doc_idx] * len(cur_text_chunks))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "354157d6-b436-4f0a-bf6e-f0a197e54c60",
      "metadata": {
        "id": "354157d6-b436-4f0a-bf6e-f0a197e54c60"
      },
      "source": [
        "### 3. Manually Construct Nodes from Text Chunks\n",
        "\n",
        "We convert each chunk into a `TextNode` object, a low-level data abstraction in LlamaIndex that stores content but also allows defining metadata + relationships with other Nodes.\n",
        "\n",
        "We inject metadata from the document into each node.\n",
        "\n",
        "This essentially replicates logic in our `SentenceSplitter`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "33b93044-3eb4-4c77-bc40-be53dffd3749",
      "metadata": {
        "id": "33b93044-3eb4-4c77-bc40-be53dffd3749"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.schema import TextNode"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "adbfcb3f-5554-4594-ae80-7236e28485aa",
      "metadata": {
        "id": "adbfcb3f-5554-4594-ae80-7236e28485aa"
      },
      "outputs": [],
      "source": [
        "nodes = []\n",
        "for idx, text_chunk in enumerate(text_chunks):\n",
        "    node = TextNode(\n",
        "        text=text_chunk,\n",
        "    )\n",
        "    src_doc_idx = doc_idxs[idx]\n",
        "    src_page = doc[src_doc_idx]\n",
        "    nodes.append(node)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "3iidPVIiwYUg",
      "metadata": {
        "id": "3iidPVIiwYUg"
      },
      "outputs": [],
      "source": [
        "print(nodes[0].metadata)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "257bb2e3-608a-4542-ba29-f29b59771a3f",
      "metadata": {
        "id": "257bb2e3-608a-4542-ba29-f29b59771a3f"
      },
      "outputs": [],
      "source": [
        "# print a sample node\n",
        "print(nodes[0].get_content(metadata_mode=\"all\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "fac468f5-870c-4486-b576-2aa6d9eaf322",
      "metadata": {
        "id": "fac468f5-870c-4486-b576-2aa6d9eaf322"
      },
      "source": [
        "### [Optional] 4. Extract Metadata from each Node\n",
        "\n",
        "We extract metadata from each Node using our Metadata extractors.\n",
        "\n",
        "This will add more metadata to each Node."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "9c369188-9cc9-4550-924e-b29d212ad057",
      "metadata": {
        "id": "9c369188-9cc9-4550-924e-b29d212ad057"
      },
      "outputs": [],
      "source": [
        "from llama_index.core.extractors import (\n",
        "    QuestionsAnsweredExtractor,\n",
        "    TitleExtractor,\n",
        ")\n",
        "from llama_index.core.ingestion import IngestionPipeline\n",
        "from llama_index.llms.openai import OpenAI\n",
        "\n",
        "llm = OpenAI(model=\"gpt-3.5-turbo\")\n",
        "\n",
        "extractors = [\n",
        "    TitleExtractor(nodes=5, llm=llm),\n",
        "    QuestionsAnsweredExtractor(questions=3, llm=llm),\n",
        "]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "f5501ffc-9bbb-4b48-9181-4e4e371e8f41",
      "metadata": {
        "id": "f5501ffc-9bbb-4b48-9181-4e4e371e8f41"
      },
      "outputs": [],
      "source": [
        "pipeline = IngestionPipeline(\n",
        "    transformations=extractors,\n",
        ")\n",
        "nodes = await pipeline.arun(nodes=nodes, in_place=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "WgbKmXr3ytPf",
      "metadata": {
        "id": "WgbKmXr3ytPf"
      },
      "outputs": [],
      "source": [
        "print(nodes[0].metadata)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "9d52522c-ffee-49d1-8651-658d70248053",
      "metadata": {
        "id": "9d52522c-ffee-49d1-8651-658d70248053"
      },
      "source": [
        "### 5. Generate Embeddings for each Node\n",
        "\n",
        "Generate document embeddings for each Node using our OpenAI embedding model (`text-embedding-ada-002`).\n",
        "\n",
        "Store these on the `embedding` property on each Node."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "6e071e36-e609-4a0c-a478-e8cfbe751cff",
      "metadata": {
        "id": "6e071e36-e609-4a0c-a478-e8cfbe751cff"
      },
      "outputs": [],
      "source": [
        "from llama_index.embeddings.openai import OpenAIEmbedding\n",
        "\n",
        "embed_model = OpenAIEmbedding()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "2047ca46-729f-4c5a-a8d7-3bc860604333",
      "metadata": {
        "id": "2047ca46-729f-4c5a-a8d7-3bc860604333"
      },
      "outputs": [],
      "source": [
        "for node in nodes:\n",
        "    node_embedding = embed_model.get_text_embedding(\n",
        "        node.get_content(metadata_mode=\"all\")\n",
        "    )\n",
        "    node.embedding = node_embedding"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "11f78014-dcca-43f5-95cb-9cfb5140b30e",
      "metadata": {
        "id": "11f78014-dcca-43f5-95cb-9cfb5140b30e"
      },
      "source": [
        "### 6. Load Nodes into a Vector Store\n",
        "\n",
        "We now insert these nodes into our `PineconeVectorStore`.\n",
        "\n",
        "**NOTE**: We skip the VectorStoreIndex abstraction, which is a higher-level abstraction that handles ingestion as well. We use `VectorStoreIndex` in the next section to fast-track retrieval/querying."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "fe34fbe4-3396-402e-8599-4b42013c3016",
      "metadata": {
        "id": "fe34fbe4-3396-402e-8599-4b42013c3016"
      },
      "outputs": [],
      "source": [
        "vector_store.add(nodes)"
      ]
    },
    {
      "cell_type": "markdown",
      "id": "74e7f8bd-d92a-40ad-8d9b-a18c04ddfca9",
      "metadata": {
        "id": "74e7f8bd-d92a-40ad-8d9b-a18c04ddfca9"
      },
      "source": [
        "## Retrieve and Query from the Vector Store\n",
        "\n",
        "Now that our ingestion is complete, we can retrieve/query this vector store.\n",
        "\n",
        "**NOTE**: We can use our high-level `VectorStoreIndex` abstraction here. See the next section to see how to define retrieval at a lower-level!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "be6a4fe1-2665-43e6-a872-8e631e31b0fd",
      "metadata": {
        "id": "be6a4fe1-2665-43e6-a872-8e631e31b0fd"
      },
      "outputs": [],
      "source": [
        "from llama_index.core import VectorStoreIndex\n",
        "from llama_index.core import StorageContext"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "0e9d2495-d4f7-469a-9cea-a5cfc401c085",
      "metadata": {
        "id": "0e9d2495-d4f7-469a-9cea-a5cfc401c085"
      },
      "outputs": [],
      "source": [
        "index = VectorStoreIndex.from_vector_store(vector_store)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "5c89e1c1-8ed1-45f5-b2a4-7c3382195693",
      "metadata": {
        "id": "5c89e1c1-8ed1-45f5-b2a4-7c3382195693"
      },
      "outputs": [],
      "source": [
        "query_engine = index.as_query_engine()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "cd6e0ddb-97c9-4f42-8843-a36a29ba3f17",
      "metadata": {
        "id": "cd6e0ddb-97c9-4f42-8843-a36a29ba3f17"
      },
      "outputs": [],
      "source": [
        "query_str = \"Can you tell me about the key concepts for safety finetuning\""
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "950cae37-7bad-44a3-be51-4154a8630818",
      "metadata": {
        "id": "950cae37-7bad-44a3-be51-4154a8630818"
      },
      "outputs": [],
      "source": [
        "response = query_engine.query(query_str)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c0b309bb-ca5a-4b15-948c-687038361c91",
      "metadata": {
        "id": "c0b309bb-ca5a-4b15-948c-687038361c91"
      },
      "outputs": [],
      "source": [
        "print(str(response))"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "basic-rag",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.14"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
